Refs:

#2650
#2197

Supersedes:

#2198

Adhoc testing script. Add a buffer without pinning it, time how long it takes to pin it. Store the time and work out the average time taken every 100 pins:

'use strict'

const last = require('it-last')
const drain = require('it-drain')

const { createController } = require('ipfsd-ctl')

async function main () {
  const ipfs = (await createController({
    type: 'go',
    ipfsBin: require('go-ipfs-dep').path(),
    ipfsHttpModule: require('ipfs-http-client'),
    disposable: false
  }))
  await ipfs.init()
  await ipfs.start()

  let times = []
  let chunk = 0

  for (let i = 0; i < 83000; i++) {
    const buf = Buffer.from(`${Math.random()}`)

    const result = await last(ipfs.api.add(buf, {
      pin: false
    }))

    const start = Date.now()

    const res = await ipfs.api.pin.add(result.cid)

    if (res[Symbol.asyncIterator]) {
      await drain(res)
    }

    const mem = process.memoryUsage()

    times.push({
      ...mem,
      elapsed: Date.now() - start
    })

    chunk++

    if (chunk === 1000) {
      const sum = times.reduce((acc, curr) => {
        acc.elapsed += curr.elapsed
        acc.rss += curr.rss
        acc.heapTotal += curr.heapTotal
        acc.heapUsed += curr.heapUsed
        acc.external += curr.external

        return acc
      }, { elapsed: 0, rss: 0, heapTotal: 0, heapUsed: 0, external: 0 })

      console.info(`${i + 1}, ${sum.elapsed / times.length}, ${sum.rss / times.length}, ${sum.heapTotal / times.length}, ${sum.heapUsed / times.length}, ${sum.external / times.length}`)

      chunk = 0
      times = []
    }
  }

  await ipfs.stop()
}

main()

Results:

10k pins, DAG vs datastore, ranges from 20-300x speedup in time taken to add a single pin:

After 100k pins, there doesn't seem to be much performance degredation in storing in the datastore whereas the DAG method degrades significantly after 8192 pins (see #2197 for discussion of that):

The next significant performance jump vs DAGs would probably be after the first layer of buckets is full - e.g. 256 buckets of 8192 pins = 2,097,152 pins. That'll probably take a bit of time to benchmark...

Next steps:

• Ensure all tests are passing
• Add concurrency to increase perf when fetching indirectly pinned CIDs for very large DAGs
• Write repo migration script
• Add docs and tests for named pins
• Make resolvePath util async iterable

That's a very cool speed improvement! Some observations:

• Double storing of CID
• Repo is no longer compatible with go-ipfs repo
• We lose the ability to share pinsets via IPFS

Double storing of CID

I guess you could only store the cid version/codec in the pin? I was thinking of changing the pin type to be an integer too, so there are definitely some improvements that can be made, this is just a first pass.

Repo is no longer compatible with go-ipfs repo

@Stebalien has talked about making a similar change to this too so it's only slightly ahead of the go-ipfs repo.

At any rate, go-ipfs is switching to badger by default which js-ipfs can't read so I'm not sure how much of a priority that is any more.

We lose the ability to share pinsets via IPFS

I guess you can't share your entire list of pins by sharing one CID, but also now do you don't have to share your entire list of pins, you can share individual ones.

Grouping multiple pins as pinsets could be added back in as a new feature, the human readable names would make this nicer to work with.

Something like:

$ ipfs pin add Qmfoo
pinned Qmfoo recursively
$ ipfs pin-set add my-super-fun-pinset Qmfoo
$ ipfs pin-set list my-super-fun-pinset
my-super-fun-pinset Qmqux
  Qmfoo
  Qmbar
  Qmbaz

You could event have the root of a pinset be an IPNS name to allow pulling updates from the network. That'd be neat.

Repo is no longer compatible with go-ipfs repo

This can be fixed :).

We lose the ability to share pinsets via IPFS

At the moment, I think this is causing strictly more harm than help. It's been 6 years and I have yet to see someone use this.

Ideally, everything would be stored in an IPLD-backed graph database. However, we aren't there yet in terms of tooling.

We could get part way there by creating an IPLD-backed datastore (datastore -> IPLD HAMT -> datastore) but that will throw away the type information.

Double storing of CID

Any reason to store the CID?

keyed by b58

Base64url? Go-ipfs, at least, now has hyper-optimized base58. However, it's still slower than base64 (and takes more space).

Questions/comments.

• How much size does this take? The definitions of these pins is now significantly larger so we should account for the overhead.
• Let's make sure to allow for arbitrary fields.
• Are the names unique?
• Given that we can't efficiently query by name, we might want to consider just calling them comments.
• We might want to consider mapping names to pins, instead of CIDs to pins. See Voker's work: Named pins & pins stored in datastore kubo#4757. The only concern here is that it significantly changes the API where as the current version just optimizes things a bit.

Some more graphs. I pinned 83k single blocks using the test script above (originally intended to be 100k but the js-dag benchmark took too long to run and I had to get on an aeroplane).

The initial hump at 8192 pins is there, then a consistent performance degradation over time. At 83k pins, js is taking 2.5s to add a pin. Go has the same degradation but it is significantly less pronounced.

The js-dag implementation stores the pinsets in memory, js-datastore does not. There is an increase in memory usage over time but it's may not be hitting the v8 gc threshold, or there's a leak somewhere...

How much size does this take? The definitions of these pins is now significantly larger so we should account for the overhead.

The sizes appear to be comparable, or perhaps they are statistically insignificant compared to the block size.

After completing the benchmark and running repo gc I see:

# js-dag
.jsipfs $  du -hs
367M	.

# go-dag
.ipfs $ du -hs
353M	.

# js-datastore
.jsipfs $ du -hs
344M	.

Let's make sure to allow for arbitrary fields.

Yes, this is the idea behind storing them CBOR encoded rather than protobufs.

Are the names unique?
Given that we can't efficiently query by name, we might want to consider just calling them comments.

Good suggestion, names are not unique so comments might be a better field name.

We might want to consider mapping names to pins, instead of CIDs to pins

If we're not going let the user query by name we probably shouldn't do this.

Any reason to store the CID?

My thinking was that by using the multihash of a block as the pin identifier (not the full CID), it becomes cheap to calculate if a given block has already been pinned (assuming the user has hashed it with the same algorithm).

The full CID is stored so we can show the user what they used to pin the block when they do a ipfs.pin.ls().

cc @hsanjuan